Add declarative benchmark matrices and shared workflows#8757
Draft
connortsui20 wants to merge 3 commits into
Draft
Add declarative benchmark matrices and shared workflows#8757connortsui20 wants to merge 3 commits into
connortsui20 wants to merge 3 commits into
Conversation
Contributor
|
Do we not use yml for the ci ones? Is this hook into there? |
connortsui20
force-pushed
the
ct/bench-declarative
branch
from
5d4d6ee to
521a932
Compare
Member
Author
|
@joseph-isaacs Yes — it's now hooked into the workflow YAML, not just a standalone module. The benchmark workflows resolve their matrix from a
Net −549 lines across the workflows. Generated by Claude Code |
connortsui20
force-pushed
the
ct/bench-declarative
branch
from
2b980ae to
51cd9f1
Compare
connortsui20
force-pushed
the
ct/bench-declarative
branch
from
ffc2b35 to
2be6840
Compare
Merging this PR will not alter performance
|
connortsui20
force-pushed
the
ct/bench-declarative
branch
from
2be6840 to
f3331ad
Compare
Define SQL benchmarks once, then resolve named profiles into the GitHub Actions matrix shape. Add the vx-bench matrix command, documentation, and focused regression coverage for selection, targets, storage fields, and preserved develop/nightly behavior. This establishes the source of truth independently of the workflow migration and addresses #4935. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Replace the duplicated full/base YAML matrices with a plan job that resolves the requested bench-orchestrator profile. Wire develop, PR, full-PR, and nightly callers to explicit profiles, derive remote paths from matrix metadata, and preserve the previous nightly iteration count. Build lance-bench whenever a resolved entry contains a Lance target so the full PR profile and its built backends stay consistent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/bench-declarative
branch
from
f3331ad to
80f1d97
Compare
Declare the two Rust micro-benchmark binaries in a small dedicated resolver and expose them through the existing matrix command. Replace the duplicated develop and PR jobs with one reusable workflow while preserving their mode-specific result handling. Keep this separate from the SQL profile migration so each change can be reviewed and landed independently. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale
Benchmark coverage was duplicated across hand-maintained SQL matrices and two nearly identical Rust micro-benchmark jobs. This made it difficult to see what CI runs and allowed the workflows to drift.
This PR moves all benchmark declarations into one deliberately obvious place:
bench_orchestrator/benchmarks.py.BENCHMARKSdeclares the SQL suites and supported targets,PROFILESdeclares the SQL coverage used by each workflow, andMICRO_BENCHMARKSlists the Rust benchmark binaries. Matrix rendering remains separate inmatrix.py.The implementation remains deliberately concrete: there is no generic predicate or tag language.
Commit structure
feat(bench-orchestrator): declare benchmark matrix profilesrefactor(ci): resolve SQL benchmark matrices from profilesrefactor(ci): share Rust micro-benchmark workflowBehavior
A semantic comparison against the removed YAML confirms that the generated
developandnightlySQL matrices are exact matches, including IDs, names, target ordering, scale-factor strings, and remote paths.The default PR SQL profile intentionally fixes the inconsistencies described in #4935:
The explicit full-PR action uses the complete declared target set. The Rust micro-benchmark consolidation preserves the existing mode-specific comparison, upload, ingest, profiling, and failure-reporting behavior.
API impact
No Vortex library APIs change.
vx-benchgains thematrixsubcommand; its existing commands are unchanged.